Systems and methods for using crowd sourcing to score online content as it relates to a belief state

ABSTRACT

A method comprising: during a first time period, obtaining one or more first content ratings of a particular content item from one or more first users, each first content rating defining a first user measure of a belief state of the particular content item; and generating a first content score for the particular content item, the first content score defining a crowd-sourced measure of the belief state of the particular content item; during a second time period, obtaining one or more second content ratings from one or more second users for the particular content item, each second content rating defining a second user measure of the belief state of the particular content item; and generating a second content score for the particular content item, the second content score defining a second crowd-sourced measure of the belief state of the particular content item; comparing the second crowd-sourced measure of the belief state of the particular content item against each of the one or more first content ratings of the particular content item to determine an expertise value for each of the one or more first users; and issuing the expertise value to each of the one or more first users.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 17/582,500, filed Jun. 28, 2022, entitled “Systems and MethodsFor Using Crowd Sourcing To Score Online Content As It Relates To ABelief State,” which is a continuation of U.S. patent application Ser.No. 17/412,150, filed Aug. 25, 2021, entitled “Systems and Methods ForUsing Crowd Sourcing To Score Online Content As It Relates To A BeliefState,” now U.S. Pat. No. 11,250,009, which claims priority to U.S.patent application Ser. No. 16/190,100, filed Nov. 13, 2018, entitled“Systems and Methods For Using Crowd Sourcing To Score Online Content AsIt Relates To A Belief State,” now U.S. Pat. No. 11,157,503, whichclaims the benefit of U.S. Provisional Patent Application Ser. No.62/586,821, filed Nov. 15, 2017, entitled “Systems and Methods forEvaluating the Veracity of News and/or Other Content,” which are herebyincorporated by reference herein.

TECHNICAL FIELD

This present disclosure relates to online content. More specifically,the present disclosure relates to systems and method for using crowdsourcing to score online content as it relates to a belief state such astruthfulness or political bias.

BACKGROUND

The news is broken and getting worse. First, the rise of the Internetand now social media has opened Pandora's box for an almost infinitenumber of content producers that has made it impossible for the averageconsumer to adequately evaluate the veracity of information they areconsuming. Fact checkers cannot keep up, and the financial incentive tokeep consumers viewing and clicking only portends for the situation toget worse. Simply put, the speed of information verification cannot keepup with pace of modern information distribution and consumption. Evenworse, nefarious actors have taken advantage of this situation toproduce “fake news” for a variety of motivations. Fake news is definedas partially or completely inaccurate content that is designed toemulate factual news without the knowledge of the consumer.

The Internet has enabled the distribution of vast amounts of informationto an incredibly large population virtually instantaneously and forcomparatively low cost. While the development of this capability hasresulted in enormous economic development and provided the great benefitof information exchange to the world, it has also exposed the samepopulation to increased risk, not the least of which emanates from fakenews. The rapid distribution of fake news can cause contagion, canmanipulate markets, can spark conflict, and can fracture strategicrelations. Most catastrophically, fake news has the potential toundermine self-government and fracture democratic institutions aroundthe world.

The current fact checking enterprise consists of independent reviewingagencies, which are only able to select a small portion of the overallcontent produced. The primary method for fact checking involves a groupof researchers associated with a particular agency (e.g., Snopes.com,FactCheck.org, etc.) reviewing an article and conducting research toverify the underlying assertions. Inherently, this process takessubstantial time and, therefore, a fact check review is typicallyreleased after the majority of consumers have already interacted withthe content. Thus, only a very small portion of content actuallyreceives external fact checking prior to consumption. Even moreproblematic is that popularity often drives the content fact checkerstarget. Thus, by definition, a large group of consumers must havealready viewed the content before the fact check even begins.Additionally, the number of fact checkers associated with each agency islimited and potentially biases the selection process for the agency.

Systems and methods are needed to avoid the potential calamity resultingfrom the “end of truth.”

SUMMARY

A claimed solution rooted in computer technology overcomes problemsspecifically arising in the realm of computer technology.

Various embodiments herein address the crisis by combiningcrowd-sourcing techniques and Bayesian probabilities to generate beliefstates of information evaluators. Various embodiments herein may utilizea news aggregator as a platform where individuals consume content items(e.g., news articles, news reports, blog articles, etc.) and can provideevaluations (or ratings) of their beliefs (e.g., on a discrete scalefrom low to high), e.g., of the veracity or political bias of thecontent items which they are consuming (e.g., reading, hearing and/orwatching). The collective set of ratings may be used to generate a groupbelief state of each of the content items.

Various embodiments herein are built around the necessity of achievingnear instantaneous “fact checking”, made possible by a large crowd ofusers who rate their beliefs of the content items. In variousembodiments, the process can begin immediately with the release of acontent item, such as the publication of a news article by a publisheron a website. By minimizing the time delay for the content item to beevaluated and providing the consumer with more information sooner, userscan better evaluate the belief state, e.g., the veracity or politicalbias, of the content item being consumed or can better filter thecontent item before it is consumed.

By leveraging the wisdom of the crowd, various embodiments effectivelymanage fake news and retain the sanctity of the “news” label.

In some embodiments, the present invention provides a system comprisingat least one hardware processor; memory storing computer instructions,the computer instructions when executed by the at least one hardwareprocessor configured to cause the system to during a first time periodthat expires upon satisfaction of a first trigger condition, obtain oneor more first content ratings of a particular content item from one ormore first users, each first content rating defining a first usermeasure of a belief state of the particular content item; and generate afirst content score for the particular content item, the first contentscore defining a crowd-sourced measure of the belief state of theparticular content item; during a second time period that expires uponsatisfaction of a second trigger condition, obtain one or more secondcontent ratings from one or more second users for the particular contentitem, each second content rating defining a second user measure of thebelief state of the particular content item; and generate a secondcontent score for the particular content item, the second content scoredefining a second crowd-sourced measure of the belief state of theparticular content item; compare the second content score of the beliefstate of the particular content item against each of the one or morefirst content ratings of the particular content item to determine anexpert score for each of the one or more first users; and issue theexpert score to each of the one or more first users.

The belief state may be truthfulness or political bias. The firstcontent score for the particular content item may be generated usingBayesian probabilities. The first trigger condition or the secondtrigger condition may include expiration of a predetermined time period.The first trigger condition or the second trigger condition may includereceiving a predetermined number of content ratings. The computerinstructions may be further configured to cause the system to, during aninitial time period, obtain one or more initial content ratings of aparticular content item from one or more initial users, each initialcontent rating defining an initial user measure of the belief state ofthe particular content item; and generate an initial content score forthe particular content item, the initial content score defining aninitial crowd-sourced measure of the belief state of the particularcontent item. Each content rating may include a discrete value between alow value and a high value. Each content rating may further include aconfidence value associated with the discrete value. The computerinstructions may further be configured to cause the system to generatethe first content score based on the expert score associated with eachfirst user.

In some embodiments, the present invention provides a method comprisingduring a first time period, obtaining one or more first content ratingsof a particular content item from one or more first users, each firstcontent rating defining a first user measure of a belief state of theparticular content item; and generating a first content score for theparticular content item, the first content score defining acrowd-sourced measure of the belief state of the particular contentitem; during a second time period, obtaining one or more second contentratings from one or more second users for the particular content item,each second content rating defining a second user measure of the beliefstate of the particular content item; and generating a second contentscore for the particular content item, the second content score defininga second crowd-sourced measure of the belief state of the particularcontent item; comparing the second crowd-sourced measure of the beliefstate of the particular content item against each of the one or morefirst content ratings of the particular content item to determine anexpertise value for each of the one or more first users; and issuing theexpertise value to each of the one or more first users.

The belief state for the method may be truthfulness or political bias.The first content score for the particular content item may be generatedusing Bayesian probabilities. The first trigger condition or the secondtrigger condition may include expiration of a predetermined time period.The first trigger condition or the second trigger condition may includereceiving a predetermined number of content ratings. The method mayfurther comprise, during an initial time period, obtaining one or moreinitial content ratings of a particular content item from one or moreinitial users, each initial content rating defining an initial usermeasure of the belief state of the particular content item; andgenerating an initial content score for the particular content item, theinitial content score defining an initial crowd-sourced measure of thebelief state of the particular content item. Each content rating mayinclude a discrete value between a low value and a high value. Eachcontent rating may further include a confidence value associated withthe discrete value. The method may further comprise generating the firstcontent score based on the expert score associated with each first user.

These and other features of the systems, methods, and non-transitorycomputer readable media disclosed herein, as well as the methods ofoperation and functions of the related elements of structure and thecombination of parts and economies of manufacture, will become moreapparent upon consideration of the following description and theappended claims with reference to the accompanying drawings, all ofwhich form a part of this specification, wherein like reference numeralsdesignate corresponding parts in the various figures. It is to beexpressly understood, however, that the drawings are for purposes ofillustration and description only and are not intended as a definitionof the limits of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a block diagram of an example content rating networksystem, according to some embodiments.

FIG. 2 depicts a block diagram of an example evaluation system,according to some embodiments.

FIG. 3 depicts a block diagram of an example extension, according tosome embodiments.

FIG. 4 depicts a block diagram of an example social media system,according to some embodiments.

FIG. 5 depicts a flow diagram of an example method of scoring contentand experts, according to some embodiments.

FIG. 6 depicts a flow diagram of an example method of generating aninitial content score of a content item, according to some embodiments.

FIG. 7 depicts a flow diagram of an example method of generating aprimary content score of a content item, according to some embodiments.

FIG. 8 depicts a flow diagram of an example method of generating anexpert score, according to some embodiments.

FIG. 9 depicts a diagram of an example computer system for implementingthe features disclosed herein, according to some embodiments.

FIG. 10 depicts an example rating panel, according to some embodiments.

FIG. 11 depicts an example social media profile, according to someembodiments.

FIG. 12 depicts an example social media activity wall, according to someembodiments.

FIG. 13 depicts an example window generated by the extension, accordingto some embodiments.

FIG. 14 depicts a mobile device having an example window generated bythe extension, according to some embodiments.

FIG. 15 depicts a flowchart of an example method of using Bayesianstatistics to generate a truthfulness score, according to someembodiments.

FIGS. 16A and 16B depict a flowchart of an example method of usingBayesian statistics to generate truthfulness scores and expert scores,according to some embodiments.

FIG. 17 depicts a flowchart of an example method of using Bayesianstatistics to generate a political bias score, according to someembodiments.

FIGS. 18A and 18B depict a flowchart of an example method of usingBayesian statistics to generate political bias scores and expert scores,according to some embodiments.

FIG. 19 depicts the types of users, according to some embodiments.

DETAILED DESCRIPTION

Various embodiments herein address the crisis by combiningcrowd-sourcing techniques and Bayesian probabilities to generate beliefstates of information evaluators. Various embodiments herein may utilizea news aggregator as a platform where individuals consume content items(e.g., news articles, news reports, blog articles, etc.) and can provideevaluations (or ratings) of their beliefs (e.g., on a discrete scalefrom low to high), e.g., of the veracity or political bias of thecontent items which they are consuming (e.g., reading, hearing and/orwatching). The collective set of ratings may be used to generate a groupbelief state of each of the content items.

Various embodiments herein are built around the necessity of achievingnear instantaneous “fact checking”, made possible by a large crowd ofusers who rate their beliefs of the content items. In variousembodiments, the process can begin immediately with the release of acontent item, such as the publication of a news article by a publisheron a website. By minimizing the time delay for the content item to beevaluated and providing the consumer with more information sooner, userscan better evaluate the belief state, e.g., the veracity or politicalbias, of the content item being consumed or can better filter thecontent item before it is consumed.

By leveraging the wisdom of the crowd, various embodiments effectivelymanage fake news and retain the sanctity of the “news” label.

FIG. 1 depicts a block diagram of an example content rating networksystem 100, according to some embodiments. The content rating networksystem 100 includes a plurality of user systems 102, each with a browser110 and an extension 112. The content rating network system 100 furtherincludes at least one server system 104 with a content evaluation system114 and a social media system 116. The content rating network system 100further includes a plurality of publisher systems 108, each with one ormore published content items 118. The plurality of user systems 102, theserver system 104 and the plurality of publisher systems 108 are coupledtogether by a computer network 106.

Each user system 102 includes a processor-based system, such as adesktop, laptop, smart pad, smart phone, etc., configured to execute thebrowser 110 and the extension 112. The browser 110, such as MicrosoftInternet Explorer™ or Apple Safari™, includes hardware, software and/orfirmware configured to navigate websites and present content items tousers.

The extension 112 includes additional hardware, software and/orfirmware, such as a plugin, configured to cooperate with the browser 110to obtain and present content scores from the server system 104, toobtain content ratings from users, to monitor user behavior with thecontent items, and to communicate the content items and/or the userbehavior to the server system 104. Additional details about theextension 112 are discussed herein.

The server system 104 includes a processor-based system, such as adesktop or laptop configured to execute the evaluation system 114 andthe social media system 116. The evaluation system 114 includeshardware, software and/or firmware configured to receive content ratingsand/or user behavior information associated with content items, togenerate content scores for the content items, to report content scoresto the users consuming the content items, to evaluate the contentratings to identify experts and generate expert scores, and to provide areward-based system to motivate users to rate content and becomeexperts. The social media system 116 includes hardware, software and/orfirmware configured to present content items to users, either inuser-specific content pages after applying filter choices set by theusers or in a generic home page after applying general filter choicesset by managers of the social media system 116. Additional details aboutthe evaluation system 114 and the social media system 116 are discussedherein.

Each publisher system 108 includes a processor-based system, such as adesktop, laptop, smart pad, smart phone, etc., configured to present theonline content items 118. Each publisher system 108 may include a systemfor a news aggregator, professional media, blogger, social media site,Twitter, Facebook, LinkedIn, YouTube, etc. Additional details of thepublisher system 108 are discussed herein.

The communication network 106 may represent one or more computernetworks or other transmission mediums. The communication network 106may provide communication between user systems 102, server systems 104and publisher systems 108 and/or other systems described herein. In someembodiments, the communication network 106 includes one or morecomputing devices, routers, cables, buses, and/or other networktopologies (e.g., mesh, and the like). The communication network 106 maybe wired and/or wireless. The communication network 106 may include theInternet, one or more wide area networks (WANs) or local area networks(LANs), public networks, private networks, IP-based networks,non-IP-based networks, and so forth.

FIG. 2 depicts a block diagram of an example evaluation system 114,according to some embodiments. The evaluation system 114 includes acontrol engine 202, a communications engine 204, a content identifierengine 206, a content information store 208, a data exchange engine 210,a content scoring engine 212, an expertise analyzer engine 214, anexpert score store 216, a content ratings store 216, an author scorestore 220, a publisher score store 222, and an expert reward engine 224.

The control engine 202 includes hardware, software and/or firmwareconfigured to manage the other components of the evaluation system 114.In some embodiments, the control engine 202 monitors trigger conditionsto determine when initial evaluators are providing content ratings thatwill be used to generate an initial content score that is allowed to bepresented to users, when primary evaluators are providing contentratings to update the initial content score to generate a dynamicallyupdating primary content score that is allowed to be presented to users,and when secondary evaluators are providing content ratings to generatea secondary content score that will be used to identify experts withinthe primary evaluators.

The communication engine 204 includes hardware, software and/or firmwareconfigured to communicate with the computer network 106. Thecommunication engine 204 may function to send requests, transmit and,receive communications, and/or otherwise provide communication with oneor a plurality of systems. In some embodiments, the communication engine204 functions to encrypt and decrypt communications. The communicationengine 204 may function to send requests to and receive data from one ormore systems through a network or a portion of a network. Depending uponimplementation-specified considerations, the communication engine 204may send requests and receive data through a connection, all or aportion of which may be wireless.

The content identifier engine 206 includes hardware, software and/orfirmware configured to identify content items. In some embodiments, thecontent identifier engine 206 begins to generate a data store of contentitems being evaluated by receiving the URL and/or a hash of the contentitem from the extension 112. In some embodiments, any content item maybe evaluated. In some embodiments, the content identifier engine 206obtains a white list of content items from the content information store208 of content items allowed to be evaluated. Notably, the hash may beused to ensure that the content item is the same content item, and hasnot been changed by the publisher or a different publisher of thecontent item. The URL and/or hash may be stored in the contentinformation store 208. That way, when a user navigates to the contentitem, the extension 112 can obtain the URL and/or the hash, can providethe identifier to the content identifier engine 206, and can receivecontent score information back from the evaluation system 114.

The data exchange engine 210 includes hardware, software and/or firmwareconfigured to facilitates information passing with the extension 112. Insome embodiments, the data exchange engine 210 is configured to obtain,process and forward content identifiers, content rating information,user comments, and/or user behavior information received from theextension 112. In some embodiments, the data exchange engine 210 isconfigured to obtain, process and forward content scores, comments andother information to the extension 112 for presentation to the users.

The content scoring engine 212 includes hardware, software and/orfirmware configured to obtain content ratings from users for contentitems as to particular belief states, e.g., truthfulness, politicalbias, etc. Content ratings may be discrete values from low to high. Insome embodiments, the content ratings may be in a 5-point scale, a10-point scale, a 20-point scale, a 100-point scale, etc. The contentratings may be presented as a spectrum, e.g., from highly conservativeto highly liberal, such that the ends of the scale define polar ends ofthe spectrum. Other mechanisms may be used to represent content ratings.In some embodiments, the user provides a content rating as to a beliefstate (e.g., truthfulness or political bias) along with his associatedconfidence level on the content rating.

The content scoring engine 212 uses Bayesian probabilities to generatecontent scores as to the different belief states based on the contentratings provided. Details of the Bayesian models are described belowwith reference to FIGS. 15, 16A, 16B, 17, 18A and 18B. Although thesystem is described as using Bayesian models, other models such asFrequentist probabilities may alternatively be used.

In some embodiments, the content scoring engine 212 may account for theexpertise of the user providing the content rating, the amount of timethat the user spent consuming the content item, whether the user visitedother sites before returning to the content item to provide the contentrating, whether the user conducted particular research regarding thecontent item prior to providing the content rating, etc.

As described in greater detail herein, the content scoring engine 212may generate an initial score provided during an initial period, e.g.,using the first few content ratings from initial users (e.g., 1, 2, 3, 4or 5). Upon the content scoring engine 212 generating the initial score,the data exchange engine 210 may provide the initial content score tothe user system 102 for presentation.

The content scoring engine 212 may continue to use Bayesianprobabilities to update the initial score with subsequent contentratings received from primary users during a primary period, e.g., untila primary trigger condition occurs, to generate a dynamically updatingprimary content score. The primary trigger condition may includeexpiration of a preliminary time period such as 1 or 2 days, receiving apredetermined number of content ratings (e.g., 100), receiving a signalfrom an external source, etc. Other alternative primary triggerconditions are possible.

In some embodiments, the content scoring engine 212 may generate asecondary score based on the content ratings received after the primarytrigger condition occurs and until a secondary trigger condition occursduring which content ratings are received from secondary users, whichare assumed to be those with a retrospective (likely better)understanding of the belief state of the content item. The secondarytrigger condition may include the expiration of a secondary time period,receiving a predetermined number of content ratings from any secondaryuser (e.g., 20 or 50), receiving a predetermined number of contentratings from users having an high expert level (e.g., 20), receiving asignal from an external source, etc. Other alternative secondary triggerconditions are possible.

In some embodiments, the content scoring engine 212 uses the secondarycontent ratings received during the secondary period to update thedynamically updating primary score which is presented to the users. Insome embodiments, the content scoring engine 212 may replace the primaryscore with the secondary score at the expiration of the secondaryperiod. In some embodiments, the content scoring engine 212 may updatethe primary score with content ratings received after the secondarytrigger condition has occurred. In some embodiments, the content scoringengine 212 may replace the primary score with the secondary score, andmay update the secondary score with the content ratings received afterthe second trigger condition has occurred.

In some embodiments, the primary period and the secondary period mayoverlap. For example, the primary period may continue as content ratingsare being provided during the secondary period. In some embodiments, theprimary period may expire when the secondary period expires.

In some embodiments, the content scoring engine 212 may ignore contentratings from users who spent less than a predetermined amount of timeconsuming the content item, suggesting that they did not give it enoughthought or are gaming the system. In some embodiments, the contentscoring engine 212 expects a Normal distribution of scores from a useracross content items, and therefore may ignore the content ratings of auser whose distribution falls outside the expectation, e.g., whoprovides the same rating across content items.

In some embodiments, the content scoring engine 212 may aggregatecontent scores to generate aggregate or entity scores. For example, thecontent scoring engine 212 may aggregate the content scores of contentitems from the same author to generate a content score of the author(e.g., the truthfulness of the author). The content scoring engine 212may aggregate the content scores of several content items published bythe same publisher to generate a content score for the publisher (e.g.,the truthfulness of the publisher). Other aggregations are possible.

The expertise analyzer engine 214 includes hardware, software and/orfirmware configured to generate expert scores of users evaluatingcontent items. In some embodiments, the expertise analyzer engine 214examines the content ratings of users who provided content ratingsduring the initial and primary periods against the secondary contentscore generated during the secondary period. Those users who wereprovided prompt and accurate content ratings during the primary contentperiod are given higher expertise points. In some embodiments, theexpertise analyzer engine 214 gives one point per prompt and accuratecontent rating over a predetermined number of content ratings, e.g., onepoint for each prompt and accurate content rating from the prior 20content ratings. In some embodiments, the expertise analyzer engine 214provides expertise points based on how close the users were to thesecondary score. For example, the expertise analyzer engine 214 may givea user 2 points if within a certain tight percentage of the secondaryscore and 1 point if the user was within a looser percentage. In someembodiments, the expertise level may be a score within a 5-point scale,a 10-point scale, a 20-point scale, a 100-point scale, etc. In someembodiments, the content ratings of users outside a current time period,e.g., 3 months, 1 year, 4 years, the Presidential term, may be deemedtoo stale to be considered.

In some embodiments, the expertise analyzer engine 214 generates ageneric expert level for each user regardless of the topic of thecontent item, on the expectation the experts want to maintain theirexpertise level and will not provide content ratings on content orbelief states with which they have little understanding. In someembodiments, the expertise analyzer engine 214 will identify theparticular topic associated with the content time, and will generateexpert scores for the experts based on their accuracy within theparticular topic area. For example, one expert may have high expertscores in politics, but low expert scores in sports. Further, in someembodiments, the expertise analyzer engine 214 will identify regions ofinterest, and will generate expert scores for the experts based on theaccuracy within each particular region. For example, one expert may havea high expert score on California-centric topics, but have a low expertscore on international topics.

In some embodiments, the expertise analyzer engine 214 may designatecontent items that are in need of retrospective evaluation. Expert usersmay select content items designated for retrospective review toevaluate, possibly for some reward. In some embodiments, the expertiseanalyzer engine 214 may only enable users who have a particularexpertise level to evaluate the content item during the secondaryperiod.

The content ratings store 218 stores the content ratings of contentitems from the users. In some embodiments, the content ratings may bestored for a predetermined time period, e.g., for 100 days. In someembodiments, the content ratings may be stored until they are determinedto be no longer relevant. The content ratings store 218 may store thecontent scores by content item identifier, e.g., by URL or hash. Thedata exchange engine 210 may obtain content scores from the contentratings store 218 to provide back to the extension 112 to present to theuser, when the user is beginning to consume a content item.

The author score store 220 stores the author scores generated based onthe content scores of the content items they authored. The author scorestore 220 may store the author scores until they are determined to be nolonger relevant. The author score store 220 may store the author scoresby author identifier, e.g., by name, user ID or email address. The dataexchange engine 210 may obtain author scores from the author score store218 to provide back to the extension 112 to present to the user, whenthe user is beginning to consume a content item by the author.

The publisher score store 222 stores the publisher scores generatedbased on the content scores of the content items they published. Thepublisher score store 222 may store the publisher scores until they aredetermined to be no longer relevant. The publisher score store 222 maystore the publisher scores by publisher identifier, e.g., by name, userID, or web address. The data exchange engine 210 may obtain publisherscores from the publisher score store 222 to provide back to theextension 112 to present to the user, when the user navigates to the webaddress, or is beginning to consume a content item published by thepublisher.

The expert rewards engine 224 includes hardware, software and/orfirmware configured to provide rewards to the experts for theirexpertise. For example, the social media system 116 may generaterevenue. The expert rewards engine 224 may track the revenue, and sharethe profits with the experts based on their expertise level andparticipation.

FIG. 3 depicts a block diagram of an example extension 112, according tosome embodiments. The extension includes a control engine 302, acommunication engine 306, a browser monitoring engine 306, a userinterface 308, and a data exchange engine 310.

The control engine 302 includes hardware, software and/or firmwareconfigured to manage the other components of the extension 112. Thecontrol engine 302 may launch the user interface 308 to present therating panel, may launch the data exchange engine 310 to request contentscores from the evaluation system 114, etc.

The communication engine 304 includes hardware, software and/or firmwareconfigured to communicate with the computer network 106. Thecommunication engine 304 may function to send requests, transmit and,receive communications, and/or otherwise provide communication with oneor a plurality of systems. In some embodiments, the communication engine304 functions to encrypt and decrypt communications. The communicationengine 304 may function to send requests to and receive data from one ormore systems through a network or a portion of a network. Depending uponimplementation-specified considerations, the communication engine 304may send requests and receive data through a connection, all or aportion of which may be wireless.

The browser monitoring engine 306 includes hardware, software and/orfirmware configured to monitor the user behavior of the user as the usernavigates websites, content items, etc. For example, the browsermonitoring engine 306 may monitor the length of time the user spends ina content items, the length of time the user spends in various parts ofthe content item, whether the user navigates to other websites beforereturning to the content item to provide a content rating, etc.

The user interface 308 includes hardware, software and/or firmwareconfigured to present a rating panel to the user. The rating panel maypresent content scores generated previously for the content item thatthe user is currently consuming. The rating panel may request contentratings from the user as to the various belief states, e.g.,truthfulness, political bias, etc.

The data exchange engine 310 may exchange information, such as thecontent scores, content ratings, user behavior data, comments, etc.,with the evaluation system 114.

FIG. 4 depicts a block diagram of an example social media system 116,according to some embodiments. The social media system 116 may include acontrol engine 402, a communication engine 404, a content selector 406,a content association engine 408, a presentation engine 410, userprofile store 412, and expert profile store 412.

The control engine 402 includes hardware, software and/or firmwareconfigured to manage the other components of the social media system116.

The communication engine 404 includes hardware, software and/or firmwareconfigured to communicate with the computer network 106. Thecommunication engine 404 may function to send requests, transmit and,receive communications, and/or otherwise provide communication with oneor a plurality of systems. In some embodiments, the communication engine404 functions to encrypt and decrypt communications. The communicationengine 404 may function to send requests to and receive data from one ormore systems through a network or a portion of a network. Depending uponimplementation-specified considerations, the communication engine 404may send requests and receive data through a connection, all or aportion of which may be wireless.

The content selector 406 includes hardware, software and/or firmwareconfigured to select content items for presentation by the social mediasystem 116. The content items may be selected based on the truthfulnessratings, political bias ratings, etc. For example, the content selector406 may only want to present content items that are highly truthful,regardless of political bias. The content selector 406 may only want topresent content items that are highly truthful and have little politicalbias. Alternatively, the content selector 406 may want to presentcontent items that are highly truthful, but divide the content itemsinto political bias categories, e.g., highly liberal content items,neutral content items, highly conservative content items. For example, auser who follows more liberal ideals may want to read content items thatare pro-Democrat or alternatively content items that are pro-Republican.In some embodiments, users may present content items to be included.Trusted users, e.g., authors with high truthfulness scores, may presentcontent items to be included.

The content association engine 408 includes hardware, software and/orfirmware configured to associate content items with the social mediasystem 116. Content association may include providing a link to thecontent item. Content association may include capturing content itemsfrom other publishers and re-publishing the content items on a websitegenerated by the social media system 116. Content association mayinclude sourcing new content generated by particular authors. In someembodiments, the content sourced by the social media system 116 mayinclude content requested from authors who have achieved certain authorscores, e.g., highly truthful authors who source content items withhighly low political bias.

The presentation engine 410 includes hardware, software and/or firmwareconfigured to present associated content to users of the social mediasystem 116. The presentation engine 410 may present a list ofcategories, from which the user selects content items. The presentationengine 410 may present content feeds which present trending contentitems on trending topics, etc. The presentation engine 410 may enableusers to search for content items using search terms, topic searches,author searches, friend searches, etc. In some embodiments, users maybecome followers of other users. For example, user A may post arecommended article, which becomes listed on the news feed of all otherusers who are friends with user A. Friends of user A who view thisarticle have the option of providing truthfulness and political biasratings on the article. In some embodiments, a user may navigate to apersonal activity wall to view content generated by connected friends.The user may click links to news articles on his wall, visit sponsorswho purchase advertisements, or post news articles which appear on hiswall and also are shared on the walls of his friends.

The user profile store 412 may store user profiles of users of thesocial media system 116. In some embodiments, the user profile store 412may store user profiles of all users who subscribe to the social mediasystem 116. The user profile store 412 may store user profiles foreveryone who visits, e.g., by maintaining a user identifier, usingcookies, etc. In some embodiments, the presentation engine 412 presentsthe user profile to the user for updating or review.

The expert profile store 412 may store the expert profiles of users ofthe social media system 116. In some embodiments, the user profile store412 only stores expert profiles for users who have generated at leastone content rating. In some embodiments, the user profile store 412 onlystores expert profiles for users who have generated at least one contentrating within a predetermined period of time, such as within the lastmonth, quarter, year, 4 years, a Presidential term, etc. In someembodiments, the presentation engine 412 presents the expert profile tothe expert for updating or review. In some embodiments, the user canenter the personal profile section and inputs personal data such ascontact information, profile photos, professional interests andaffiliations, academic interests and affiliations, and personalinterests. The user can also manage his personal connections, such asadding or removing friends.

FIG. 5 depicts a flow diagram of an example method 500 of scoringcontent items and experts, according to some embodiments. Method 500begins in step 502 with the evaluation system 114 obtaining initialcontent ratings as to a belief state during an initial period frominitial users. In step 504, the evaluation system 114 awaits an initialtrigger condition that marks the end of the initial period. For example,the initial trigger condition may include having received apredetermined number of initial content ratings (e.g., 1, 5, 10 or 100)as to the belief state, a manual event, a predetermined length of time,etc. Upon satisfying the initial trigger condition, the evaluationsystem 114 generates an initial content score, which will be provided tosubsequent users. In some embodiments, prior to obtaining an initialcontent score, the evaluation system 114 will not provide any contentscore to the user.

In step 508, during a primary period, the evaluation system 114 obtainsprimary content ratings as to the belief state from primary users. Instep 510, the evaluation system 114 awaits a primary trigger conditionthat marks the end of the primary period. The primary trigger conditionmay include a predetermined number of primary content ratings (e.g.,100), a predetermined period of time (e.g., 1 or 2 days), apredetermined date and time (e.g., midnight on day 1), an external event(e.g., sundown or the end of a show), a manual event, etc. In step 512,the evaluation system 114 generates the primary content score based oncontent ratings received from the initial users and primary users (whoprovided scores during the initial period and the primary period).

In step 514, during a secondary period, the evaluation system 114obtains secondary content ratings from secondary users. In step 516, theevaluation system 114 awaits a secondary trigger condition that marksthe end of the secondary period. The secondary trigger condition mayinclude a predetermined number of primary content ratings (e.g., 50), apredetermined period of time (e.g., 1 day), a predetermined date andtime (e.g., midnight on day 2), an external event (e.g., sundown or theend of a show), a manual event, etc. In step 518, the evaluation system114 generates a secondary content score. In step 520, the evaluationsystem 114 generates expert scores based on a comparison between thesecondary content scores and the primary ratings of the primary usersand between the secondary content scores and the initial ratings of theinitial users.

FIG. 6 depicts a flow diagram of an example method 600 of generating aninitial content score of a particular content item, according to someembodiments. The method 600 begins in step 602 with the extension 112(e.g., the browser monitoring engine 306) monitoring the browser forcontent items being consumed by the user. In step 604, the extension 112(e.g., the data exchange engine 310) sends the content identifier (e.g.,the URL or hash of the content item) to the evaluation system 114 (e.g.,the data exchange engine 210). In step 606, the evaluation system 114(e.g., the content identifier engine 206) determines whether the contentidentifier is associated with content to be scored and whether it has ascore. If there is no content score, then in step 608, the evaluationsystem 114 (e.g., the data exchange engine 210) sends a request for acontent rating to the extension 112 (e.g., the data exchange engine310). In step 610, the extension 112 (e.g., the user interface 308)presents the content rating request to the user. In step 612, theextension 112 (e.g., the browser monitoring engine 306) monitors theuser behavior as the user consumes the content item. In step 614, theextension 112 (e.g., the data exchange engine 310) sends the userbehavior information and the content rating to the evaluation system 114(e.g., the data exchange engine 210). In step 616, the evaluation system114 (e.g., the content scoring engine 212) applies the user behaviorinformation and applies Bayesian probabilities the content rating fromthis user the content ratings from other uses, to generate an initialcontent score. In step 618, the evaluation system 114 (e.g., the contentratings store 218) stores the initial content ratings and the initialcontent score in the content ratings store 218.

FIG. 7 depicts a flow diagram of an example method 700 of generating aprimary content score of a particular content item, according to someembodiments. The method 700 begins in step 702 with the extension 112(e.g., the browser monitoring engine 306) monitoring the browser forcontent items being consumed by the user. In step 704, the extension 112(e.g., the data exchange engine 310) sends the content identifier (e.g.,the URL or hash of the content item) to the evaluation system 114 (e.g.,the data exchange engine 210). In step 706, the evaluation system 114(e.g., the content identifier engine 206) determines whether the contentidentifier is associated with content to be scored and whether it hasalready received a score. If it has already received a score, then instep 708, the evaluation system 114 (e.g., the data exchange engine 210)sends an current content score to the extension 112 (e.g., the dataexchange engine 310). In step 710, the extension 112 (e.g., the userinterface 308) presents the current content score to the user and instep 712 requests a content rating from the user. In step 714, theextension 112 (e.g., the browser monitoring engine 306) monitors theuser behavior as the user consumes the content item. In step 716, theextension 112 (e.g., the data exchange engine 310) sends the userbehavior information and the content rating to the evaluation system 114(e.g., the data exchange engine 210). In step 718, the evaluation system114 (e.g., the content scoring engine 212) applies the user behaviorinformation and applies Bayesian probabilities on the content ratingfrom this user and on the current content score, to update the currentcontent score to a new content score. In step 720, the evaluation system114 (e.g., the content ratings store 218) stores the content rating andthe new content score in the content ratings store 218.

FIG. 8 depicts a flow diagram of an example method 800 of generating anexpert score, according to some embodiments. The method 800 begins instep 802 with the evaluation system 114 (e.g., the control engine 202 orthe expertise analyzer engine 214) determining that the primary periodhas expired. In step 804, the extension 112 (e.g., the browsermonitoring engine 306) monitoring the browser for content items beingconsumed by the user during the secondary period. In step 806, theextension 112 (e.g., the data exchange engine 310) sends the contentidentifier (e.g., the URL or hash of the content item) to the evaluationsystem 114 (e.g., the data exchange engine 210). In step 808, theevaluation system 114 (e.g., the content identifier engine 206) uses thecontent identifier to identify the content to be scored and to retrievethe current primary content score. In step 810, the evaluation system114 (e.g., the data exchange engine 210) sends the primary content scoreto the extension 112 (e.g., the data exchange engine 310). In step 812,the extension 112 (e.g., the user interface 308) presents the primarycontent score to the user. In step 814, the extension 112 (e.g., theuser interface 308) requests the user to provide a content rating. Instep 816, the extension 112 (e.g., the browser monitoring engine 306)monitors the user behavior as the user consumes the content item. Instep 818, the extension 112 (e.g., the data exchange engine 310) sendsthe user behavior information and the content rating to the evaluationsystem 114 (e.g., the data exchange engine 210). In step 820, theevaluation system 114 (e.g., the content scoring engine 212) applies theuser behavior information and applies Bayesian probabilities on thecontent rating from this user, and on the content ratings from otherusers during the secondary period, to generate a secondary contentscore. In step 822, the evaluation system 114 (e.g., the expertiseanalyzer engine 214) compares the secondary content score against priorcontent ratings of the initial users and primary users who providedcontent ratings during the initial period and the primary period toevaluate expertise. In step 824, the evaluation system 114 (e.g., theexpertise analyzer engine 214) generates expert scores based on thecomparison. In step 826, the evaluation system 114 (e.g., the contentratings store 218) stores the expert stores in the expert score store216.

FIG. 9 depicts a block diagram of an example of a computing device 900.Any of the systems 102, 104 and/or 108, and the communication network106 may comprise an instance of one or more computing devices 900. Thecomputing device 900 comprises a hardware processor 904, memory 906,storage 908, an input device 910, a communication network interface 912,and an output device 914 communicatively coupled to a communicationchannel 916. The processor 904 is configured to execute executableinstructions (e.g., programs). In some embodiments, the processor 904comprises circuitry or any processor capable of processing theexecutable instructions.

The memory 906 stores data. Some examples of memory 906 include storagedevices, such as RAM, ROM, RAM cache, virtual memory, etc. In variousembodiments, working data is stored within the memory 906. The datawithin the memory 906 may be cleared or ultimately transferred to thestorage 908.

The storage 908 includes any storage device configured to retrieve andstore data. Some examples of the storage 908 include flash drives, harddrives, optical drives, cloud storage, and/or magnetic tape. Each of thememory system 906 and the storage system 908 comprises acomputer-readable medium, which stores instructions or programsexecutable by processor 904. The distinction between memory 906 andstorage 908 has been blurring, so memory 906 and storage 908 should betreated interchangeably.

The input device 910 includes any device that receives data (e.g., mouseand keyboard). The output device 914 includes any device that presentsdata (e.g., a speaker and/or display).

The communication network interface 912 may be coupled to a network(e.g., network 106) via the link 918. The communication networkinterface 912 may support communication over an Ethernet connection, aserial connection, a parallel connection, and/or an ATA connection. Thecommunication network interface 912 may also support wirelesscommunication (e.g., 802.11 a/b/g/n, WiMax, LTE, WiFi) or wiredcommunication.

The hardware elements of the computing device 900 are not limited tothose depicted in FIG. 9. A computing device 902 may comprise more orless hardware, software and/or firmware components than those depicted(e.g., drivers, operating systems, touch screens, biometric analyzers,and/or the like). In certain circumstances, the storage 908, inputdevice 910, and output device 914 may be optional. Further, hardwareelements may share functionality and still be within various embodimentsdescribed herein. In one example, encoding and/or decoding may beperformed by the processor 904 and/or a co-processor located on a GPU(e.g., Nvidia GPU).

FIG. 10 depicts an example rating panel 1000, according to someembodiments. The extension 112 may present the rating panel 1000adjacent to or on top of the content item associated with it. In someembodiments, the rating panel 1000 may request the user to input one ormore belief states associated with the content item. As shown, theexample rating panel 1000 has a truthfulness belief state interface 1002requesting a truthfulness belief state rating and has a political biasbelief state interface 1004 requesting a political bias belief staterating. As shown, the user placed a star 1006 to the high side of thetruthfulness interface 1002 representing an opinion that the contentitem is highly truthful, and the user placed a star 1008 to the low sideof the political bias interface 1004 representing an opinion that thecontent item has low political bias. In other embodiments, the user maybe able to rate the belief state in different interfaces.

FIG. 11 depicts an example social media profile 1100, according to someembodiments. As stated above, the social media system 116 may generate asocial media site for users, to support content item filtering andconsumption and to support expert development. As shown, the socialmedia profile 1100 may include a search bar 1102, a news tab 1104, anactivity wall 1106, a profile tab 1108, an employment history field1110, an education history field 1112, a user identifier field 1114 andan expert score field 1124. The user identifier field 1114 may include astatus level indicator 1116, a profile picture 1118, a total ratingsindicator 1120, and an expert score indicator 1122. In addition tocontent item aggregation, the social media side may also host a socialmedia environment to link users together as a community. Similar toother social media communities, the social media system 116 may allowusers to create a profile for free and to share personal information onan individualized profile page. The profile page may display informationabout the history of the user's ratings, and a total number of ratingsthat a user has conducted. The profile page may also assign an aggregateexpert score to the user based on the accuracy of the user's ratings.Based on these two metrics, the user may be assigned a status level. Thestatus level may recognize those users who have rated often andaccurately. The profile page may offer the user an opportunity todisplay a photo along with employment and education information. In someembodiments, the expert score may also be broken down by topic tospecifically recognize the expertise of the user.

FIG. 12 depicts an example social media activity wall 1200, according tosome embodiments. In addition to some of the elements of FIG. 11 (suchas the picture, name, job title, status level, total ratings, and expertscore), the social media activity wall 1200 may display in a field 1202recent comments by other users linked to this specific user and recentratings of content items identified as of interest to this user. Thesocial activity wall 1200 may also display in a field 1204 content itemstrending among the users that are performing ratings, other users linkedto this user, or topics identified as of interest to this user.

FIG. 13 depicts an example window 1300 generated by the extension 112,according to some embodiments. The window 1300 includes a URL field 1302identifying the URL location of a content items, a field 1306 presentingthe content item (e.g., a news article), and a rating panel 1308 similarto the rating panel 1000. The window 1304 further includes an extensiontab (labeled as “Add-in”) 1304, which when selected causes the extension112 to present the rating panel 1308. Much like other extensions, theextension tab 1304 may ride on top of the browser window 1300, and willnot affect the basic function of that browser 110. If the user isreading the news article contained in field 1306, the extension 112 willidentify the content item using the URL contained in the URL field 1302,will display the rating panel 1308 for that particular news article, andwill offer the user to rate the news article (even if it has not beenpreviously rated).

FIG. 14 depicts an example mobile device 1400 having a mobile extensionwindow 1402 generated by the extension 112, according to someembodiments. The window 1402 may include a field 1404 displaying acontent item (e.g., a feature story). Adjacent to the field 1404, theextension 112 may generate a rating panel 1406 to request user to ratethe content item as to one or more belief states. The window 1402 maypresent similar content items in a more simplified, mobile friendlydisplay environment. Users may also be able to provide rating feedbackusing the rating panel 1406. Topic listings (Topic 1, Topic 2, Topic 3and Topic 4) will automatically be prioritized based on user definedpreferences combined with online activity.

FIG. 15 depicts a flowchart of an example method 1500 of using Bayesianprobabilities to generate a prompt truthfulness score, according to someembodiments. As described above, the collective set of evaluationsrepresents the user population's prior belief state of the content item.Various embodiments use Bayesian updating methods to modify the beliefstate over two time windows: a primary period and secondary period. Inthe primary period (nominally hours-to-days, representing promptevaluations of a newly published content item), as ratings aresubmitted, the evaluation system 114 uses Bayesian updating to updatethe collective belief state regarding the truthfulness of the contentitem. The method 1500 begins in step 1502 with the evaluation system 114diffusing a prior distribution on truthfulness. The evaluation system114 in step 1504 receives a truthfulness rating from a first user and instep 1506 uses Bayesian updating to update the distribution ontruthfulness. The evaluation system 114 in step 1508 receives atruthfulness rating from a second user and in step 1510 uses Bayesianupdating to update again the distribution on truthfulness. Theevaluation system 114 repeats this n times. As shown, the evaluationsystem 114 in step 1512 receives a truthfulness rating from an nth userand in step 1514 uses Bayesian updating to update again the distributionon truthfulness.

FIGS. 16A and 16B depict a flowchart of an example method 1600 of usingBayesian probabilities to generate truthfulness scores and expertscores, according to some embodiments. Various embodiments use Bayesianupdating techniques to modify the belief state (e.g., truthfulness) overtwo time windows: a primary period and a secondary period. Theevaluation system 114 may compare the truthfulness ratings submitted inthe primary period (nominally hours-to-days, representing promptevaluations of a newly published content item), against truthfulnessratings provided in the secondary period to evaluate expertise.

As shown, the method 1600 includes a truthfulness evaluation processthat repeats for N content items (e.g., articles). The method 1600begins a primary period for a first content item (e.g., article 1) instep 1602A, during which the evaluation system 114 receives an initialsampling of user ratings to establish a prior belief state (e.g., atruthfulness state) from the initial sampling. The method 1600 continuesin step 1604A during which the evaluation system 114 uses Bayesianupdating techniques to update the initial belief state with additionaluser ratings received during the primary period. The method 1600continues until the conclusion of the primary period in step 1606A bythe evaluation system 114 generating a belief-state score (e.g., atruthfulness score). The method 1600 begins secondary period in step1608A, during which the evaluation system 114 receives a set ofretrospective ratings from users as to the belief state and generates aretrospective belief-state score from the retrospective sampling. Themethod 1600 continues until the conclusion of the secondary period instep 1610A by the evaluation system 114 uses the retrospectivebelief-state score to evaluate users to identify experts from the userswho rated the belief state of the content item during the initial periodand primary period. The method 1600 continues in steps 1602B, 1604B,1606B, 1608B and 1610B for a second content item (e.g., article 2). Themethod 1600 repeats for N content items, the evaluation of the Nthcontent item (e.g., article N) being shown in steps 1602N, 1604N, 1606N,1608N and 1610N. Further details of method 1600 are provided below.

Primary Time Period

Starting with a diffuse prior (either a variant of a Normal distributionor beta(1,1)) as users provide ratings on the truthfulness of a contentitem, the belief state distribution is updated in a sequential fashionin the following manner:

Suppose user n believes that the truthfulness of a content item is ratedas a value

$\frac{C}{N} \in {( {0,1} ).}$

The evaluation system 114 uses Bayesian updating to formulate theposterior distribution on the truthfulness of the content item, given adiffuse prior, as given by

${{P( {{\pi ❘C},N} )} = \frac{{P( {C,{N❘\pi}} )}{P(\pi)}}{P( {C,N} )}},$

where π is the truthfulness parameter. The likelihood function, or theprobability of observing the truthfulness rating

$\frac{C}{N}$

given π is defined as

${{P( {C,{N❘\pi}} )} = {\begin{pmatrix}N \\C\end{pmatrix}{\pi^{C}( {1 - \pi} )}^{N - C}}},$

since the evaluation system 114 treats the

$\frac{C}{N}$

rating as a proxy for the total proportion of times that C truthfulcontent items will be observed among N total content items. Therefore,the evaluation system 114 treats the likelihood function as a binomialdistribution function. The marginal likelihood, or evidence, is given by

P(C, N)=∫₀ ¹P(C, N|π)P(π)dπ.

Assuming the prior distribution can be characterized with a Betadistribution, the evaluation system 114 defines the prior distributionas

${{P(\pi)} = {{{Beta}( {\alpha,\beta} )} = \frac{{\pi^{\alpha - 1}( {1 - \pi} )}^{\beta - 1}}{B( {\alpha,\beta} )}}},$

where

${B( {\alpha,\beta} )} = {\frac{{\Gamma(\alpha)}{\Gamma(\beta)}}{\Gamma( {\alpha + \beta} )}.}$

Then, by substituting the components of the Bayesian expression, theevaluation system 114 obtains

${P( {{\pi ❘C},N} )} = {{\frac{\lbrack {\begin{pmatrix}N \\C\end{pmatrix}{\pi^{C}( {1 - \pi} )}^{N - C}} \rbrack\lbrack \frac{{\pi^{\alpha - 1}( {1 - \pi} )}^{\beta - 1}}{B( {\alpha,\beta} )} \rbrack}{\int_{0}^{1}{{P( {C,{N❘\pi}} )}{P(\pi)}d\pi}} \propto {{\pi^{C}( {1 - \pi} )}^{N - C}{\pi^{\alpha - 1}( {1 - \pi} )}^{\beta - 1}}} = {{\pi^{C + \alpha - 1}( {1 - \pi} )}^{N - C + \beta - 1}.}}$

The evaluation system 114 transforms the expression of proportionalityto an expression of equality by using a constant K to represent theproportionality factor to obtain

P(π|C, N)=Kπ ^(C+α−1)(1−π)^(N−C+β−1).

Since the posterior must integrate to 1, the evaluation system 114modifies the posterior distribution to

${{P( { \pi \middle| C ,\ N} )} = {\begin{pmatrix}{N + \alpha + \beta - 2} \\{C + \alpha}\end{pmatrix} \times {\pi^{C + \alpha - 1}( {1 - \pi} )}^{N - C + \beta - 1}}},$

which corresponds to a Beta distribution with updated parameters, asgiven by

Beta(α+C, β+N−C).

Given the input of another user's rating, denoted

${\frac{C^{\prime}}{N^{\prime}} \in ( {0,1} )},$

the evaluation system 114 defines the revised posterior distribution as

${{P( {{\pi ❘C},N,C^{\prime},N^{\prime}} )} = {\begin{pmatrix}{N + N^{\prime} + \alpha + \beta - 2} \\{C + C^{\prime} + \alpha}\end{pmatrix} \times {\pi^{C + C^{\prime} + \alpha - 1}( {1 - \pi} )}^{N + N^{\prime} - {({C + C^{\prime}})} + \beta - 1}}},$

which corresponds to a Beta distribution with updated parameters:

Beta(α+C+C′, β+N+N′−C−C′)

Now, given this technique of updating the prior belief distribution intoa posterior belief distribution, the evaluation system 114 cangeneralize the model for a sequence of T updates in the followingmanner:

For time t₀<T, let the prior distribution be

Beta(α+C _(t) ₀ , β+N _(t) ₀ −C _(t) ₀ ).

Then, the posterior distribution is

${Beta}{( {{\alpha + {\sum\limits_{i = 1}^{T}C_{t_{i}}}},{\beta + {\sum\limits_{i = 1}^{T}N_{t_{i}}} - {\sum\limits_{i = 1}^{T}C_{t_{i}}}}} ).}$

By implementing this technique over n user ratings on content itemtruthfulness, the evaluation system 114 can establish a posteriordistribution during the prompt evaluation period. The evaluation system114 reports this sequentially updated belief distribution to thecommunity of users, so that they will have an understanding of thecollective belief of the content item. The evaluation system 114 mayapply the following intuition:

-   -   Belief distributions with higher expected values represent a        collective belief that the content item is more truthful than        content items with relatively lower belief distribution expected        values.    -   Belief distributions with lower expected values represent a        collective belief that the content item is less truthful than        content items with relatively higher belief distribution        expected values.

Secondary Time Period

In the secondary period (nominally days to weeks after an article hasinitially been published, representing retrospective evaluations ofarticles), a subset of the user population that did not originallyevaluate the content item may be asked to provide their evaluations onthe content item. Their ratings may represent a belief state on thecontent item treated as a proxy for an accurate belief state on thetruthfulness of the content item. Given this belief state, individualswhose prompt belief ratings closely match the collective's retrospectivebelief state are identified as “expert evaluators,” and as a result theevaluation system 114 may weight their evaluations more heavily onsubsequent prompt evaluations. The evaluation system 114 may employseveral techniques of assessing whether a prompt evaluation constitutesa match with a posterior evaluation. The simplest technique is byexpected value matching:

Match criteria: User n truth rating_(prompt)=EV[beliefdistribution_(retrospective)].

If a user's truthfulness rating during the prompt evaluation periodmatches the expected value of the retrospective belief distribution,then this is considered a match. Alternately, the evaluation system 114can apply a gradient function that results in a “degree of match” (DOM)as a function of the difference between the user's prompt rating and theexpected value of the retrospective belief distribution, according to

${DOM}_{n} = {\frac{1}{\begin{matrix}{{{User}n{truth}{rating}_{prompt}} -} \\{{EV}\lbrack {{retrospective}{belief}{distribution}} \rbrack}\end{matrix}}.}$

The degree of match carries a conditional value according to

${DOM}_{n} = \{ {\begin{matrix}\frac{1}{\begin{matrix}{{{User}n{truth}{rating}_{prompt}} -} \\{{EV}\lbrack {{belief}{distribution}_{retrospective}} \rbrack}\end{matrix}} \\{0,}\end{matrix},\begin{matrix}{{match}{criteria}{satisfied}} \\\begin{matrix} \\{otherwise}\end{matrix}\end{matrix}} $

The evaluation system 114 may use the DOM_(n) values in subsequentprompt evaluations by weighting the ratings from high DOM_(n) valueusers more heavily than users with low DOM_(n) values.

By employing an ongoing, continuous flow of prompt and retrospectivetruthfulness evaluations of many content items over time, the evaluationsystem 114 may (1) establish a belief on the truthfulness of contentitems based upon crowd-sourced evaluations, and (2) identify expertevaluators from among the crowd to further improve the efficacy of fakenews detection.

FIG. 17 depicts a flowchart of an example method 1700 of using Bayesianstatistics to generate a prompt political bias score, according to someembodiments. As described above, the collective set of evaluationsrepresents the user population's prior belief state of the content item.Various embodiments use Bayesian updating methods to modify the beliefstate over two time windows: a primary period and a secondary period. Inthe primary period (nominally hours-to-days, representing promptevaluations of a newly published content item), as ratings aresubmitted, the evaluation system 114 uses Bayesian updating to updatethe collective belief state regarding the political bias of the contentitem. The method 1700 begins in step 1702 with the evaluation system 114diffusing a prior distribution on political bias. The evaluation system114 in step 1704 receives a political bias rating from a first user andin step 1706 uses Bayesian updating to update the distribution onpolitical bias. The evaluation system 114 in step 1708 receives apolitical bias rating from a second user and in step 1710 uses Bayesianupdating to update again the distribution on political bias. Theevaluation system 114 repeats this n times. As shown, the evaluationsystem 114 in step 1712 receives a political bias rating from an n^(th)user and in step 1714 uses Bayesian updating to update again thedistribution on political bias.

FIGS. 18A and 18B depict a flowchart of an example method 1800 of usingBayesian statistics to generate political bias scores and expert scores,according to some embodiments. Various embodiments use Bayesian updatingtechniques to modify the belief state (e.g., political bias) over twotime windows: a primary period and a secondary period. The evaluationsystem 114 may compare the political bias ratings submitted in theprimary period (nominally hours-to-days, representing prompt evaluationsof a newly published content item), against political bias ratingsprovided in the secondary period to evaluate expertise.

As shown, the method 1800 includes a political bias evaluation processthat repeats for N content items (e.g., articles). The method 1800begins a primary period for a first content item (e.g., article 1) instep 1802A, during which the evaluation system 114 receives a promptsampling of user ratings to establish a prior belief state (e.g., apolitical bias state) from the prompt sampling. The method 1800continues in step 1804A during which the evaluation system 114 usesBayesian updating techniques to update the prompt belief state withadditional user ratings received during the primary period. The method1800 continues until the conclusion of the primary period in step 1806Aby the evaluation system 114 generating a belief-state score (e.g., apolitical bias score). The method 1800 begins a secondary period in step1808A, during which the evaluation system 114 receives a set ofretrospective ratings from users as to the belief state and generates aretrospective belief-state score from the retrospective sampling. Themethod 1800 continues at the conclusion of the secondary period in step1810A by the evaluation system 114 uses the retrospective belief-statescore to evaluate users to identify experts from the users who rated thebelief state of the content item during the initial period and primaryperiod. The method 1800 continues in steps 1802B, 1804B, 1806B, 1808Band 1810B for a second content item (e.g., article 2). The method 1800repeats for N content items, the evaluation of the Nth content item(e.g., article N) being shown in steps 1802N, 1804N, 1806N, 1808N and1810N. Further details of method 1800 are provided below.

Primary Time Period

Starting with a diffuse prior (either a variant of a Normal distributionor beta(1,1)) as users provide evaluations on the political bias of acontent item, the belief state distribution is updated in a sequentialfashion in the following manner:

Suppose user n believes that the political bias of a content item israted as a value

$\frac{C}{N} \in {( {0,1} ).}$

For example:

$\begin{pmatrix}\begin{matrix}{\frac{C}{N} > 0.5} \\{\frac{C}{N} = 0.5}\end{matrix} \\{\frac{C}{N} < 0.5}\end{pmatrix} = {\begin{pmatrix}\begin{matrix}{conservative} \\{neutral}\end{matrix} \\{liberal}\end{pmatrix}.}$

Then, the evaluation system 114 uses Bayesian updating to formulate theposterior distribution on the political bias of the content item, givena diffuse prior, as given by

${{P( {{\pi ❘C},N} )} = \frac{{P( {C,{N❘\pi}} )}{P(\pi)}}{P( {C,N} )}},$

where π is the political bias parameter. The likelihood function, or theprobability of observing the political bias rating

$\frac{C}{N}$

given π is defined as

${{P( {C,{N❘\pi}} )} = {\begin{pmatrix}N \\C\end{pmatrix}{\pi^{C}( {1 - \pi} )}^{N - C}}},$

since the evaluation system 114 treats the

$\frac{C}{N}$

rating as a proxy for the total proportion of times that C unbiasedcontent items will be observed among N total articles. Therefore, theevaluation system 114 treats the likelihood function as a binomialdistribution function. The marginal likelihood, or evidence, is given by

P(C, N)=∫₀ ¹P(C, N|π)P(π)dπ.

Assuming the prior distribution can be characterized with a Betadistribution, the evaluation system 114 defines the prior distributionas

${{P(\pi)} = {{{Beta}( {\alpha,\beta} )} = \frac{{\pi^{\alpha - 1}( {1 - \pi} )}^{\beta - 1}}{B( {\alpha,\beta} )}}},$

where

${B( {\alpha,\beta} )} = {\frac{{\Gamma(\alpha)}{\Gamma(\beta)}}{\Gamma( {\alpha + \beta} )}.}$

Then, by substituting the components of the Bayesian expression, theevaluation system 114 obtains

${P( {{\pi ❘C},N} )} = {{\frac{\lbrack {\begin{pmatrix}N \\C\end{pmatrix}{\pi^{C}( {1 - \pi} )}^{N - C}} \rbrack\lbrack \frac{{\pi^{\alpha - 1}( {1 - \pi} )}^{\beta - 1}}{B( {\alpha,\beta} )} \rbrack}{\int_{0}^{1}{{P( {C,{N❘\pi}} )}{P(\pi)}d\pi}} \propto {{\pi^{C}( {1 - \pi} )}^{N - C}{\pi^{\alpha - 1}( {1 - \pi} )}^{\beta - 1}}} = {{\pi^{C + \alpha - 1}( {1 - \pi} )}^{N - C + \beta - 1}.}}$

The evaluation system 114 transforms this expression of proportionalityto an expression of equality by using a constant K to represent theproportionality factor to obtain

P(π|C, N)=Kπ ^(C+α−1)(1−π)^(N−C+β−1).

Since the posterior must integrate to 1, the evaluation system 114modifies the posterior distribution to

${{P( {{\pi ❘C},N} )} = {\begin{pmatrix}{N + \alpha + \beta - 2} \\{C + \alpha}\end{pmatrix} \times {\pi^{C + \alpha - 1}( {1 - \pi} )}^{N - C + \beta - 1}}},$

which corresponds to a Beta distribution with updated parameters, asgiven by

Beta(α+C, β+N−C).

Given the input of another user's rating, denoted

${\frac{C^{\prime}}{N^{\prime}} \in ( {0,1} )},$

the evaluation system 114 defines the revised posterior distribution as

${{P( {{\pi ❘C},N,C^{\prime},N^{\prime}} )} = {\begin{pmatrix}{N + N^{\prime} + \alpha + \beta - 2} \\{C + C^{\prime} + \alpha}\end{pmatrix} \times {\pi^{C + C^{\prime} + \alpha - 1}( {1 - \pi} )}^{N + N^{\prime} - {({C + C^{\prime}})} + \beta - 1}}},$

which corresponds to a Beta distribution with updated parameters:

Beta(α+C+C′, β+N+N′−C−C′).

Now, given this technique of updating the prior belief distribution intoa posterior belief distribution, the evaluation system 114 generalizesthe model for a sequence of T updates in the following manner:

For time t₀<T, let the prior distribution be

Beta(α+C _(t) ₀ , β+N _(t) ₀ −C _(t) ₀ ).

Then the posterior distribution is

${Beta}{( {{\alpha + {\sum\limits_{i = 1}^{T}C_{t_{i}}}},{\beta + {\sum\limits_{i = 1}^{T}N_{t_{i}}} - {\sum\limits_{i = 1}^{T}C_{t_{i}}}}} ).}$

By implementing this technique over n user ratings on content itempolitical bias, the evaluation system 114 establishes a posteriordistribution during the prompt evaluation period. The evaluation system114 reports this sequentially updated belief distribution to thecommunity of users, so that they will have an understanding of thecollective belief of the content item. The evaluation system 114 appliesthe following intuition:

-   -   Belief distributions with higher expected values represent a        collective belief that the content item is more        conservative-leaning than content items with relatively lower        belief distribution expected values.    -   Belief distributions with lower expected values represent a        collective belief that the content item is more liberal-leaning        than content items with relatively higher belief distribution        expected values.

Secondary Time Period

In the secondary period (nominally days to weeks after an article hasinitially been published, representing retrospective evaluations ofarticles), a subset of the user population that did not originallyevaluate the content item is asked to provide their evaluations on thecontent item. Their ratings may represent a belief state on the contentitem treated as a proxy for an accurate belief state on the politicalbias of the content item. Given this belief state, individuals whoseprompt beliefs closely match the collective's retrospective belief stateare identified as “expert evaluators,” and as a result the evaluationsystem 114 may weight their evaluations more heavily on subsequentprompt evaluations. The evaluation system 114 may employ severaltechniques of assessing whether a prompt evaluation constitutes a matchwith a posterior evaluation. The simplest technique is by expected valuematching:

Match criteria: User n political bias rating_(prompt)=EV[retrospectivebelief distribution]

If a user's political bias rating during the prompt evaluation periodmatches the expected value of the retrospective belief distribution,then this may be considered a match. Alternately, the evaluation system114 can apply a gradient function that results in a “degree of match”(DOM) as a function of the difference between the user's prompt ratingand the expected value of the retrospective belief distribution,according to

${DOM}_{n} = \frac{1}{\begin{matrix}{{{User}n{political}{bias}{rating}_{prompt}} -} \\{{EV}\lbrack {{belief}{distribution}_{retrospective}} \rbrack}\end{matrix}}$

The degree of match carries a conditional value according to

${DOM}_{n} = \{ \begin{matrix}{\frac{1}{\begin{matrix}{{{User}n{political}{bias}{rating}_{prompt}} -} \\{{EV}\lbrack {{belief}{distribution}_{retrospective}} \rbrack}\end{matrix}},} & {{match}{criteria}{satisfied}} \\{0,} & {otherwise}\end{matrix} $

The evaluation system 114 may use the DOM_(n) values in subsequentprompt evaluations by weighting the ratings from high DOM_(n) valueusers more heavily than users with low DOM_(n) values.

By employing an ongoing, continuous flow of prompt and retrospectivepolitical bias evaluations of many content items over time, theevaluation system 114 may (1) establish a belief on the political biasof content items based upon crowd-sourced evaluations, and (2) identifyexpert evaluators from among the crowd to further improve the efficacyof a political bias detector.

FIG. 19 depicts the types of users, according to some embodiments. Thetypes of users include passive users 1900, a subsection of which areactive users 1902, a subsection of which become expert users 1904.

Passive Users are an integral part of the community and may constitutethe bulk of the user base. A passive user may be someone who consumescontent items but does not actively provide content ratings. In someembodiments, although passive users are welcome to rate content items,they are not required to do so. In some embodiments, a social mediaaccount on the social media system 116 is not needed to provide ratings.In some embodiments, for a user to receive credit for providing contentratings, a social media account on the social media system 116 must beestablished. The passive user receives the benefit of receivingimmediate and clear feedback concerning the veracity and/or politicalbias and/or other belief state of a content item. In some embodiments,even without an account, a passive user may obtain personalization ofthe experience based on prior activity on the social media website. Forexample, the social media system 116 may provide recommendations ofcontent items based on prior reading interests.

Active users may be defined as those people who have established anaccount on the social media system 116 or users who have at any timeprovided a content rating, who recently provided a content rating, whoregularly provide content ratings, and/or the like. In some embodiments,active users may be people who contribute content items to the socialmedia website. The social media system 116 may provide opportunity foractive users to connect with other active users with similar interestsand/or expertise, to share information about themselves and theirbackgrounds, and/or to provide comments on the articles in addition tothe numerical ratings. In some embodiments, the social media system 116may enable active users to comment on content items that are at issueand suggest corrections to make the articles more accurate. In someembodiments, the social media system 116 may enable the active user toadd a narrative to a comment section associated with a content item,e.g., to cite specific evidence of an content item's inaccuracyincluding links to other possibly related content items.

Experts are active users who have proven their expertise as promptevaluators of content items, possibly divided by topic area or possiblygenerally across all areas. In some embodiments, experts are categorizedbased on area of expertise and a record of the expert scores may bemaintained of the experts based on the accuracy of their rating record.In some embodiments, each of the expert scores will be time adjusted toensure that more recent ratings account for a larger component of thescores. This will ensure continuous quality control of the experts,encourage new experts to join the community, and simultaneouslyencourage existing experts to remain engaged. Experts will be able to,if desired, leverage their expert status in other areas of their lives,potentially for professional gain. For example, a person identified as apolitical expert may parlay the recognition of that expertise into theirown opportunities providing commentary. The same can be said for someonewho is identified as an expert sports prognosticator. In order toprovide effective incentives for experts to contribute to thiscommunity, some embodiments of the social media system 116 may sharerevenue across the expert community based on a set of rules. In someembodiments, the experts that rate most often and most accurately willreceive the greatest compensation. The compensation need not be linearso the most expert can earn significantly more than marginal experts.

It will be appreciated that an “engine,” “system,” “datastore,” and/or“database” may comprise software, hardware, firmware, and/or circuitry.In one example, one or more software programs comprising instructionscapable of being executable by a processor may perform one or more ofthe functions of the engines, datastores, databases, or systemsdescribed herein. In another example, circuitry may perform the same orsimilar functions. Alternative embodiments may comprise more, less, orfunctionally equivalent engines, systems, datastores, or databases, andstill be within the scope of present embodiments. For example, thefunctionality of the various systems, engines, datastores, and/ordatabases may be combined or divided differently. The datastore ordatabase may include cloud storage. It will further be appreciated thatthe term “or,” as used herein, may be construed in either an inclusiveor exclusive sense. Moreover, plural instances may be provided forresources, operations, or structures described herein as a singleinstance.

The datastores described herein may be any suitable structure (e.g., anactive database, a relational database, a self-referential database, atable, a matrix, an array, a flat file, a documented-oriented storagesystem, a non-relational No-SQL system, and the like), and may becloud-based or otherwise.

The systems, methods, engines, datastores, and/or databases describedherein may be at least partially processor-implemented, with aparticular processor or processors being an example of hardware. Forexample, at least some of the operations of a method may be performed byone or more processors or processor-implemented engines. Moreover, theone or more processors may also operate to support performance of therelevant operations in a “cloud computing” environment or as a “softwareas a service” (SaaS). For example, at least some of the operations maybe performed by a group of computers (as examples of machines includingprocessors), with these operations being accessible via a network (e.g.,the Internet) and via one or more appropriate interfaces (e.g., anApplication Program Interface (API)).

The performance of certain of the operations may be distributed amongthe processors, not only residing within a single machine, but deployedacross a number of machines. In some example embodiments, the processorsor processor-implemented engines may be located in a single geographiclocation (e.g., within a home environment, an office environment, or aserver farm). In other example embodiments, the processors orprocessor-implemented engines may be distributed across a number ofgeographic locations.

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures andfunctionality presented as separate components in example configurationsmay be implemented as a combined structure or component. Similarly,structures and functionality presented as a single component may beimplemented as separate components. These and other variations,modifications, additions, and improvements fall within the scope of thesubject matter herein.

The embodiments herein are described above with reference to examples.It will be apparent to those skilled in the art that variousmodifications may be made and other embodiments may be used withoutdeparting from the broader scope of the teachings herein. Therefore,these and other variations upon the example embodiments are intended tobe covered.

1. A system comprising: at least one hardware processor; and memorystoring computer instructions configured to assist in having contentitems that are presented by a web browser or an application evaluated,the computer instructions when executed by the at least one hardwareprocessor being configured to cause the system to during a first timeperiod, obtain first content evaluations of a particular content itemfrom first users, each first content evaluation defining a first userestimation of a belief state of subject matter content in the particularcontent item; during a second time period, after the first time period,review each of the first content evaluations of the particular contentitem from the first users; during the second time period, generate anexpert score for each of the first users who provided the first contentevaluations of the particular content item based on the review of thefirst content evaluations; assist in displaying the first contentevaluations in association with the particular content item; issue theexpert score to each of the first users; and elevate an influentialcharacteristic of the first users having an expert score higher than acertain threshold when providing another content evaluation of anothercontent item during another time period.
 2. The system of claim 1,wherein the belief state is truthfulness of the subject matter contentin the particular content item.
 3. The system of claim 1, wherein thebelief state is political bias of the subject matter content in theparticular content item.
 4. The system of claim 1, wherein each contentevaluation includes a discrete value between a low value and a highvalue.
 5. The system of claim 4, wherein each content evaluation furtherincludes a confidence value associated with the discrete value.
 6. Thesystem of claim 1, wherein the computer instructions are furtherconfigured to cause the system to: capture first user behaviorinformation of each first user of the first users; and evaluate based onthe first user behavior information whether to accept each first contentevaluation of the first content evaluations from each first user of thefirst users.
 7. The system of claim 1, wherein the computer instructionsare further configured to cause the system to: generate an author scoreof an author of the particular content item based on at least a portionof the first content evaluations.
 8. The system of claim 7, wherein thecomputer instructions are further configured to cause the system to:generate the author score of the author of the particular content itembased further on the expert scores of the first users of the at least aportion of the first content evaluations.
 9. The system of claim 1,wherein the computer instructions are further configured to cause thesystem to: generate a publisher score of a publisher of the particularcontent item based on at least a portion of the first contentevaluations.
 10. The system of claim 9, wherein the computerinstructions are further configured to cause the system to: generate thepublisher score of the publisher of the particular content item basedfurther on the expert scores of the first users of the at least aportion of the first content evaluations.
 11. The system of claim 1,wherein the expert score corresponds to a particular topic of contentitems.
 12. The system of claim 1, wherein the expert score is a generalexpert score across all topics of content items.
 13. The system of claim1, wherein the computer instructions are further configured to cause thesystem to: discard the expert score when it is deemed stale.
 14. Amethod by a processor executing computer instructions configured toassist in having content items that are presented by a web browser or anapplication evaluated, the method comprising: during a first timeperiod, obtaining first content evaluations of a particular content itemfrom first users, each first content evaluation defining a first userestimation of a belief state of subject matter content in the particularcontent item; during a second time period, after the first time period,review each of the first content evaluations of the particular contentitem from the first users; during the second time period, generate anexpert score for each of the first users who provided the first contentevaluations of the particular content item based on the review of thefirst content evaluations; assisting in displaying the first contentevaluations in association with the particular content item; issuing theexpert score to each of the first users; and elevating an influentialcharacteristic of the first users having an expert score higher than acertain threshold when providing another content evaluation of anothercontent item during another time period.
 15. The method of claim 14,wherein the belief state is truthfulness of the subject matter contentin the particular content item.
 16. The method of claim 14, wherein thebelief state is political bias of the subject matter content in theparticular content item.
 17. The method of claim 14, wherein eachcontent evaluation includes a discrete value between a low value and ahigh value.
 18. The method of claim 17, wherein each content evaluationfurther includes a confidence value associated with the discrete value.19. The method of claim 14, further comprising: capturing first userbehavior information of each first user of the first users; andevaluating based on the first user behavior information whether toaccept each first content evaluation of the first content evaluationsfrom each first user of the first users.
 20. The method of claim 14,further comprising: generating an author score of an author of theparticular content item based on at least a portion of the first contentevaluations.
 21. The method of claim 20, further comprising: generatingthe author score of the author of the particular content item basedfurther on the expert scores of the first users of the at least aportion of the first content evaluations.
 22. The method of claim 14,further comprising: generating a publisher score of a publisher of theparticular content item based on at least a portion of the first contentevaluations.
 23. The method of claim 22, further comprising: generatingthe publisher score of the publisher of the particular content itembased further on the expert scores of the first users of the at least aportion of the first content evaluations.
 24. The method of claim 14,wherein the expert score corresponds to a particular topic of contentitems.
 25. The method of claim 14, wherein the expert score is a generalexpert score across all topics of content items.
 26. The method of claim14, further comprising: discarding the expert score when it is deemedstale.